Anatomical Sketch Understanding: Recognizing Explicit and Implicit Structure

OBJECTIVE Sketching is ubiquitous in medicine. Physicians commonly use sketches as part of their note taking in patient records and to help convey diagnoses and treatments to patients. Medical students frequently use sketches to help them think through clinical problems in individual and group problem solving. Applications ranging from automated patient records to medical education software could benefit greatly from the richer and more natural interfaces that would be enabled by the ability to understand sketches. In this paper we take the first steps toward developing a system that can understand anatomical sketches. METHODS Understanding an anatomical sketch requires the ability to recognize what anatomical structure has been sketched and from what view (e.g. parietal view of the brain), as well as to identify the anatomical parts and their locations in the sketch (e.g. parts of the brain), even if they have not been explicitly drawn. We present novel algorithms for sketch recognition and for part identification. We evaluate the accuracy of the recognition algorithm on sketches obtained from medical students. We evaluate the part identification algorithm by comparing its results to the judgment of an experienced physician. RESULTS The sketch recognition algorithm achieves a recognition accuracy of 75.5%, far above the baseline random classification accuracy of 6.7%. Comparison of the results of the part identification algorithm with the judgment of an experienced physician shows close agreement in terms of location, orientation, size, and shape of the identified parts. CONCLUSIONS The performance of our prototype in terms of accuracy and running time provides strong evidence that development of robust sketch understanding systems for medical domains is an attainable goal. Further work needs to be done to extend the approach to sketches containing multiple and partial anatomical structures, as well as to be able to interpret sketch annotations.